{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backtesting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back testing is basically lingo for testing the performance of a strategy (or a prediction model) on historical data. The trader would need to consider various things before back testing especially with regards to the historical data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Historical Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many free and commercial historical data that can be accessed. We will focus mostly on google's historical data and talk about issues such as data splitting and dividend adjustment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Splitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Companies may have their stock split N to 1. So previous a stock would have been \\$140 per share on Aug 1 2015 and they may have decided to split it 2 to 1 thus the price that shows up on Yahoo would be $70 per share on Aug 2 2015. In such a case if we analyze the data without adjusting for splitting then the algorithm may think that there was a decrease in that time... which is false. So we need to adjust the previous stock price accordingly. Because the split happend as a 2 to 1, we may divide the previous values by 2 but this isn't the only adjustment that needs to be made. All dividens also needs to be accounted for. Let's generate some data below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        close       date\n",
      "0  141.764052 2015-08-01\n",
      "1  140.400157 2015-08-02\n",
      "2  140.978738 2015-08-03\n",
      "3  142.240893 2015-08-04\n",
      "4  141.867558 2015-08-05\n",
      "5   68.045444 2015-08-06\n",
      "6   71.900177 2015-08-07\n",
      "7   69.697286 2015-08-08\n",
      "8   69.793562 2015-08-09\n",
      "9   70.821197 2015-08-10\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "\n",
    "# generate fake data\n",
    "date_range = pd.Series(pd.date_range('2015-08-01', periods=10))\n",
    "df = pd.DataFrame({\n",
    "    'date': pd.concat([date_range]),\n",
    "    'close': np.concatenate((np.random.normal(140, 1, 5),np.random.normal(70, 2, 5)),axis = 0)\n",
    "}).reset_index(drop=True)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in the dataframe on Aug 6th the stock split and we would need to get the dividend information in order to adjust according to the dividends earned after Aug 5th. Here is a random generation of dividen data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        date  dividend  prevCLose\n",
      "0 2015-12-28  0.437752  68.936788\n",
      "1 2016-07-19  0.447843  69.175456\n",
      "2 2015-09-04  0.496649  69.193082\n",
      "3 2015-10-30  0.515120  66.339943\n",
      "4 2015-11-02  0.295963  68.608330\n",
      "5 2017-03-13  0.576377  70.493532\n",
      "6 2016-04-15  0.734996  73.051915\n",
      "7 2017-07-03  0.542516  68.454456\n",
      "8 2017-06-22  0.675151  71.764113\n",
      "9 2017-03-03  0.441837  67.494813\n"
     ]
    }
   ],
   "source": [
    "date_range = pd.Series(pd.date_range('2015-08-01', '2017-08-01'))\n",
    "\n",
    "div = pd.DataFrame({\n",
    "    'date': np.random.choice(date_range[1:-1], size = 10, replace=False),\n",
    "    'prevCLose' : np.random.normal(70, 2, 10),\n",
    "    'dividend': np.random.beta(10,10,10)\n",
    "}).reset_index(drop=True)\n",
    "print(div)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the dividend information we need to calculate the multiplier information as follows:\n",
    "$$ \\text{Multiplier} = \\frac{\\text{Previous Closing Price} - \\text{Dividend}}{\\text{Previous Closing Price}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "div['Multiplier'] = (div['prevCLose'] - div['dividend'])/div['prevCLose']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        date  dividend  prevCLose  Multiplier\n",
      "0 2015-12-28  0.437752  68.936788    0.993650\n",
      "1 2016-07-19  0.447843  69.175456    0.993526\n",
      "2 2015-09-04  0.496649  69.193082    0.992822\n",
      "3 2015-10-30  0.515120  66.339943    0.992235\n",
      "4 2015-11-02  0.295963  68.608330    0.995686\n",
      "5 2017-03-13  0.576377  70.493532    0.991824\n",
      "6 2016-04-15  0.734996  73.051915    0.989939\n",
      "7 2017-07-03  0.542516  68.454456    0.992075\n",
      "8 2017-06-22  0.675151  71.764113    0.990592\n",
      "9 2017-03-03  0.441837  67.494813    0.993454\n"
     ]
    }
   ],
   "source": [
    "print(div)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.46411004688410229"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.prod(div['Multiplier'])*0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The product of the multipliers after the split date would need to be used as the adjustment for the previous time points before stock splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>close</th>\n",
       "      <th>date</th>\n",
       "      <th>adj_close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>141.764052</td>\n",
       "      <td>2015-08-01</td>\n",
       "      <td>65.794121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>140.400157</td>\n",
       "      <td>2015-08-02</td>\n",
       "      <td>65.161124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>140.978738</td>\n",
       "      <td>2015-08-03</td>\n",
       "      <td>65.429649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>142.240893</td>\n",
       "      <td>2015-08-04</td>\n",
       "      <td>66.015428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>141.867558</td>\n",
       "      <td>2015-08-05</td>\n",
       "      <td>65.842159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>68.045444</td>\n",
       "      <td>2015-08-06</td>\n",
       "      <td>68.045444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>71.900177</td>\n",
       "      <td>2015-08-07</td>\n",
       "      <td>71.900177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>69.697286</td>\n",
       "      <td>2015-08-08</td>\n",
       "      <td>69.697286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>69.793562</td>\n",
       "      <td>2015-08-09</td>\n",
       "      <td>69.793562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>70.821197</td>\n",
       "      <td>2015-08-10</td>\n",
       "      <td>70.821197</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        close       date  adj_close\n",
       "0  141.764052 2015-08-01  65.794121\n",
       "1  140.400157 2015-08-02  65.161124\n",
       "2  140.978738 2015-08-03  65.429649\n",
       "3  142.240893 2015-08-04  66.015428\n",
       "4  141.867558 2015-08-05  65.842159\n",
       "5   68.045444 2015-08-06  68.045444\n",
       "6   71.900177 2015-08-07  71.900177\n",
       "7   69.697286 2015-08-08  69.697286\n",
       "8   69.793562 2015-08-09  69.793562\n",
       "9   70.821197 2015-08-10  70.821197"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for x in range(len(df)):\n",
    "    if df.loc[x, 'date'] < pd.datetime(2015,8,6):\n",
    "        df.loc[x,'adj_close'] = df.loc[x,'close']*np.prod(div['Multiplier'])*0.5\n",
    "    else:\n",
    "        df.loc[x,'adj_close'] = df.loc[x,'close']\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus as we can see the adjusted prices pre Aug 06 will be adjusted accordingly and then the algorithm should be run on this data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Survivor ship bias in data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since data that contains survivorship of companies (i.e. companies that have gone backrupt or delisted are expensive, we need to either 1. collect current data to make sure we include survivorship of companies from here on out or 2. make lots of money and buy the proprietary data that includes such information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing to be cognizant is the effect of survivorship bias. If we focus on getting low priced stocks, it is likely that those companies are going bankrupt soon... so read into not just the price but other qualitative information (news and etc) to determine if that is the case and stay away from those stocks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High and Low Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the high and low prices are far more noisier compared to the open and close prices. There are many possibilities that may cause the reliability of the high and low price to be worse than that of the open and close prices. FOr example, a order may not have been filled due to the fact that a very small order was transacted at the high, or the execution could have occured ona a market to which the order wasn't routed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically, the discrepancies of the open and close prices usually have less impact on backtest performance than errors in the high and low prices, since high and low price errors almost always inflate the backtest returns. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moral of this section is to do at least a simple error check of the dataset with high, low, open and closeing prices such that various combinations of daily returns can be calculated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance Measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are various performance measures that people may use and we will not go over all of them. Most of the time return based measures will require explanation regarding the denominator and numerator. Thus it is relatively straight forward to use the Sharp ratio and drawdown values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In terms of using the Sharp ratio, the risk free rate from the returns of a dollar-neutral portfolio (i.e. long short equal capital) should not be subtracted. The reason is because a dollar-neutral portfolio is self-financing (i.e. the cash earned from selling short pays for the purchase of long securities so we assume that the financing cost is small and can be neglected for backtesting). Meanwhile, the margin balance that needs to be maintained earns a credit interest close to the risk-free rate $r_F$ thus the excess return used in calculating the Sharp Ratio would be the strategy return $R$ plus the return from the self-financing $r_F$ subtratced by the credit interested $r_F$ would be $R + r_F - r_F = R$. \n",
    "In general, only subtracte the risk-free rate from the $R$ when calculating the Sharp ratio if there are financing costs that are involved specifically for long-only strategies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another aspect to facilitate comparison across strategies is to annualize the Sharp ratio. If we are using monthly returns, the average annual return is 12 * the monthly return. The standard deviation would be $\\sqrt{12}$ according to Sharpe. Here is the general equation:\n",
    "$$ \\text{Annualized Sharpe Ratio} = \\sqrt{N_T} \\times \\text{Sharpe Ratio Based on T}$$\n",
    "Where T is the trading period."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example if we trade only for 6.5 hours a day for 252 days then $N_T = 252 \\times 6.5 = 1638$ so the Annulized Sharpe Ratio would be $\\sqrt{1638} \\times R/s$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further calculation and example using bitcoin is in [Definition and Formulas](https://github.com/ck2136/QuantTrade/blob/master/Definitions%20and%20Formulas.ipynb) Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## COmmon Backtesting Pitfalls to Avoid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've already talked about how survivorship bias can inflate strategy performance. There are two other common mistakes to avoid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look-Ahead Bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If our strategy for buying involves buying at 1% of the day's low, this introduces the *look-ahead bias*. Basically we wouldn't know what the day's low until we had all values when the market closes. The way to avoid this is to use *lagged* historical data for calculating signals at every opportunity. This basically translates into conducting analyses with data up to the close of the *previous* trading period only. In `R` or `Python` this would involve using a lag function to run aanalses on a series of columns that would produce a *lagged* column for regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Programmatically to check for look-ahead bias, we would create two datasets where one is using the original data and running the regression then truncating the most recent N observations while a new data set would also have the most recent N data removed but only after truncation through *lagging* do we run the regression. We then align the two predicted positions based on the regression and see if there were any difference. If there are then there is look-ahead bias. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data-snooping bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This has to do with over fitting the data with numerous parameters and going back in time more than 10 years than may not be relevant. The goal is to get a parsimonimous model that really only involves parameters that will be highly predictive of future positions. The author of QT indicates that using aroudn 5 parameters given highly dependent dataset would normally suffice in creating moving-average models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to mitigate data snooping bias there are some considerations below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample Size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The greater then number of parameter used, the more data points required for optimizing the model. The statistical community is in constant development regarding the optimal sample size requirement for time-series prediction models. Check [Hanke and Wichern](https://www.amazon.com/dp/0132301202/?tag=stackoverflow17-20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to robustly check whether or not the predictions are really going to be useful, there needs to be an out-of-sample testing dataset that can be used to test the prediction model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More robust optimization of the model would employ moving optimizaiton of the parameters. The parameters are constantly adapting to the changing historical data and data snooping in that regard would be reduced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sensitivity Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once optimization has been achieved using the various ideas described above, varying the conditions to see how the results of the prediction model changes on the training and testing data. If there is considerable difference in the performance masures such that only the optimal condition is apporpriate, that means that the strategy may not be robust therefore may result in poor predictions down the line. Important variations that need to tried:\n",
    "- Simplify the model\n",
    "    - Eliminate conditions, constraints, and parameters to make the model more parsimonious\n",
    "- Set up trading strategy such that multiple variation of the final model (which has been tested to have good performance) will be used such that trading capital should be divided among them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transaction Costs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also one should take into consideration the transaction cost when thinking about developing indicators for buying and selling. This will again be implemented in the other notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strategy Refinement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pair trading: Buying and selling stocks from similar stock universe\n",
    "- Exclusion of stocks or groups of stocks from universe"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
